In making driving decisions, typical autonomous vehicle (AV) systems take account of objects—such as other vehicles and obstacles—that the AV system knows are in the environment of the AV, either because sensor systems on the AV observe the objects, or because the objects are identified by maps or other data sources. For making driving decisions, the AV systems may maintain world models that include the objects known to be in the AV's environment. Challenges to good driving decisions also arise from vehicles and obstacles that the AV cannot perceive and does not otherwise know to exist based on available data.